1. Technical Field
The present invention relates to techniques for digitally capturing and processing high dynamic range still and video images of a scene using relatively low resolution image detectors, and more particularly, to techniques capable of adaptively capturing and processing such images.
2. Background Art
In digital photography and video imaging, it is often necessary to capture images of a scene which includes areas of greatly disparate luminance. An extreme example lies in the imaging of a solar eclipse, where a nearly black object, the moon, is directly adjacent an extraordinarily bright object, the sun. Other more common examples include photographing a person with the sun in the background, videotaping a sporting event in the late afternoon hours where shadows are evident, and capturing images of a fireworks display.
In each of these examples, the range of luminance values needed to fully capture the scene is high, ranging from near black to extremely bright. As a typical image capture device, such as a charge-coupled device (“CCD”) operates with eight bits of resolution, only 256 luminance values are available to span the entire brightness spectrum in such circumstances, and often yield a poor images. While higher dynamic range image sensors, such as sixteen bit CCDs, are commercially available, they come at an increased cost which may be undesirable, and regardless of scene brightness operate within a fixed dynamic range.
Accordingly, there have been several attempts to provide a technique for capturing and processing high dynamic range still and video images of a scene using relatively low resolution image detectors, such as the above-mentioned eight bit CCD. Those attempts may be generally grouped into sequential exposure change methods, techniques using multiple image detectors, approaches using multiple sensor elements within a pixel, adaptive pixel exposure techniques and methods employing spatially varying pixel exposures.
The most obvious approach is to sequentially capture multiple images of the same scene using different exposures. The exposure for each image is controlled by either varying the F-number of the imaging optics or the exposure time of the image detector. A high exposure image will be saturated in the bright scene areas but capture the dark regions well. In contrast, a low exposure image will have less saturation in bright regions but end up being too dark and noisy in the dark areas. The complementary nature of these images allows one to combine them into a single high dynamic range. Such an approach has been employed in Japanese Patent No. 08-223491 of H. Doi, et al. entitled “Image Sensor” (1986) and by others since then. The sequential exposure charge approach was subsequently extended by using acquired images to compute a radiometric response function of the imaging system, e.g., as described in T. Mitsunaga et al., “Radiometric Self Calibration,” 1 Proc. of Computer Vision and Pattern Recognition '99,” pp. 374-380 (1999).
Unfortunately, sequential exposure change techniques are inherently suited only to static scenes, and the imaging system, scene objects and their respective radiance levels must remain constant during the sequential capture of images under different exposures. If the images can be captured in quick succession, they can be merged to obtained high dynamic range images at a reasonable frame-rate, as described in U.S. Pat. No. 5,144,442 to R. Ginosar et al. However, the above noted constants must be maintained.
The stationary scene restriction faced by sequential capture may be remedied by using multiple imaging systems. This approach is described in the above mentioned Japanese Patent No. 08-223491, as well as in Japanese Patent 08-340486 to K. Saito (1996) and elsewhere. Beam splitters are used to generate multiple copies of the optical image of the scene. Each copy is detected by an image detector whose exposure is preset by using an optical attenuator or by changing the exposure time of the detector. While this approach is capable of producing high dynamic range images in real time, with both scene objects and the imaging system free to move during the capture process, a disadvantage is that this approach is expensive as it requires multiple image detectors, precision optics for the alignment of all the acquired images and additional hardware for the capture and processing of multiple images.
Another approach to high dynamic range imaging uses a different CCD design, where multiple sensor elements lie within a pixel. In this approach, each detector cell includes two sensing elements, i.e., potential wells, of different sizes and hence sensitivity. When the detector is exposed to the scene, two measurements are made within each cell and they are combined on-chip before the image is read out. Such an approach is proposed in U.S. Pat. No. 5,789,737 to R. A. Street, and elsewhere. However, this technique is also expensive as it requires a sophisticated detector to be fabricated. In addition, spatial resolution is reduced by a factor of two since the two potential wells take up the same space as two pixels in a conventional image detector. Further, the technique is forced to use a simple combining technique for the outputs of the two wells as it is done on-chip.
A further approach to high dynamic range imaging has been proposed in V. Brajovic et al., “A Sorting Image Sensor: An Example of Massively Parallel Intensity-to-Time Processing for Low-Latency Computational Sensors,” Proc. of IEEE Conference on Robotics and Automation, pp. 1638-1643 (1996). There, a particular solid state image sensor is developed where each pixel on the device includes a computational element that measures the time it takes to attain full potential well capacity. Since the full-well capacity is the same for all pixels, the time to achieve it is proportional to image radiance. The recorded time values are read out and converted to a high dynamic range image. This approach faces the challenge of scaling to high resolution while keeping fabrication costs under control. In addition, since exposure times can be large in dark scene regions, the method is expected to be more susceptible to motion blur.
Finally, an approach to high dynamic range imaging using spatial variation of pixel sensitivity has been disclosed in S. K. Nayar et al., “High dynamic range imaging: Spatially varying Pixel exposures,” Proc. of Computer Vision and Pattern Recognition '00 (2000). Different (fixed) sensitivities are assigned to neighboring pixels on the image detector. An important feature here is the simultaneous sampling of spatial dimensions as well as the exposure dimension of image radiance: when a pixel is saturated in the acquired image, it is likely to have a neighbor that is not, and when a pixel produces zero brightness, it is likely to have a neighbor that produces non-zero brightness. Unfortunately, a trade-off of spatial resolution of an image is made in order to improve brightness resolution, or dynamic range. Moreover, the fixed pixel exposure values do not provide the ability to measure each image brightness with the best precision possible. Accordingly, there remains a need for an imaging technique which optimizes the dynamic range of a captured image of a scene, regardless of the dynamic range of the image sensor employed.